Assignment #2 - Gradient Domain Fusion

Yu-Chen Lin

I. Description

Overview

In this homework, we try to blend images to achieve natural image synthesis. The challenge is that directly cropping and pasting the source image into the target image can lead to easily identified artifacts. For example, if, after cropping, our source image still includes some background noise, then it will be easy for people to identify the inconsistency of that background noise if we only do direct image blending.

Solution

A possible solution that we implement in the homework is called Poisson blending. The algorithm aims to utilize the gradient part in two steps. The first objective is to make the gradient of generated pixel values close to the gradient of the source image. The second objective is to ensure the gradient of the generated pixel values in the boundary close to the gradient of the target image. This way, the gradient behavior of the source image can be preserved, and the generated pixels also take the gradient of the target image into consideration.

Feedback

This is an interesting project because even though the algorithm is not too complicated, the blending results actually look very realistic. Nowadays, people tend to use more complicated generative models to do things. In my opinion, though the traditional methods are not state-of-the-art anymore, they are still meaningful to learning the concept behind them. Further, compared to large generative models, they can run really fast with effective results.

II. Toy Problem

Following the instructions, I can successfully reconstruct the original image by gradient-based. At first, it is magical for me that we only need one-pixel intensity with other gradient-based constraints to recover the whole image. After some thinking process, I believe the algorithm makes sense because the intensity constraint of one pixel will affect the other pixels in the image through the gradient constraint.

III. Favorite Blending Result

The source image (the left one) was taken when I was doing skydiving in Dubai. I would like to make it look like space diving. Thus, I chose the target image (the right one) that shows a spacecraft and the universe.

The first thing I did was to manually crop the image to only include the region of interest and then downsample the image to reduce the computational cost.

The result after Poisson blending is too dark because the objective only cares about the gradient.

To address this problem, observing that the objective functions have some dependency on the target intensity, I try to make the target image brighter by adjusting the parameter gamma. The results seem to be more reasonable. However, there are still some artifacts around the source region.

I do not do something very fancy but just select the masking region more carefully, which leads to my final results.

IV. More Results

Shark & Scuba Diving

It will be terrifying if a shark is on your back when you are doing scuba diving. I want to use image blending to achieve this. The source image is chosen as a fierce-looking shark. The target image may be a diver who just feels excited when seeing the world of the ocean.

The naive blending is already good. However, the color of the shark may appear somewhat artificial. The Poisson blending helps to make the shark integrate into the oceanic environment.

Dinosaur & Zoo

Though Dinosaurs are extint nowadays, we are still fascinated by these creatures. If little children can see dinosaurs in the zoom, that will definitely be a great memory. Therefore, I picked a source image of a dinosaur and a target image that a child is looking at the elephant.

The result looks okay but still seems not quite realistic in some parts.

  • The style of the dinosaur seems not to be aligned with the background of the zoo. We can select a dinosaur that has a similar style, but it will be time-consuming. This issue should be considered as a limitation of Poisson blending.
  • The sense of the space does not look right. For the elephant, its feet are blocked by the fence. However, the dinosaur seems to be out of the fence in its lower part, but its upper part appears inside of the fence. This point is also a limitation of Poisson blending.
  • Similar to the previous point, after applying Poisson blending, the grass on the dinosaur makes the image look even less realistic than the naive blending result. The objective tends to consider the background noise of the target image, which causes such an artificial result.

    • V. Bells & Whistles - Mixed Gradients

    The instructions suggest blending a picture of writing on a plain background onto another image. Thus, I chose a spring couplet as the target image and my own writing as the source image.

    The mixed gradients blending seems to work very well. It successfully blends my writing into the spring couplet.